Linstor add support for incremental nasbackup - #14194

Description

This PR adds incremental NAS backup support for KVM VMs whose disks live on raw block-device storage such as LINSTOR/DRBD.

#13877 (backport of #13074) implements incremental NAS backups with QEMU dirty bitmaps and libvirt checkpoints. Bitmap persistence is a qcow2-only feature, so allVolumesOnCheckpointCapableStorage() excludes raw block devices and every LINSTOR VM stays pinned to full-only backups regardless of the configured cadence. On top of that, the push-mode full backup of a raw device writes a fully allocated qcow2 which then needs a second qemu-img convert pass to reclaim space.

This PR adds a content-based backup path driven by libvirt pull mode and routes VMs on content-diff-capable storage (currently LINSTOR) to it:

nasbackup.sh – new content backup method

The script gets a --method option next to -M|--mode: qcow2 (default, the existing push-mode path with dirty bitmaps) or content. With --method content, libvirt is asked for a pull-mode backup: it starts an NBD server exposing a point-in-time view of each disk, held consistent by copy-before-write into a scratch block device per disk. The backup is then derived from that export:

• full: qemu-img convert from the NBD export straight into a sparse qcow2 in a single pass. This replaces the bloated push-mode output and its re-convert pass.
• incremental: a qcow2 overlay backed by the NBD export, then a safe qemu-img rebase onto the parent backup. This merges in exactly the clusters that differ and yields a delta qcow2 backed by the parent, with a relative backing path. The chain has the same shape as a bitmap incremental, so the existing restore and cascade-delete logic applies unchanged.

A missing or short parent list degrades to a full and emits INCREMENTAL_FALLBACK=true, mirroring the push-mode fallback. Also handles the raw /dev/drbdN device paths of long-running VMs the same way the push/stopped paths do (via get_linstor_uuid_from_device).

The scratch devices are passed with --scratch-devs, one per disk; libvirt formats them as qcow2 (<scratch dev=.../>). The NBD socket and the backup XML live in a small work directory under /var/tmp.

A pull-mode job never ends on its own, so an EXIT/INT/TERM/HUP trap ends it (via abort_backup_job() from #14305), removes the work directory and cleans up on any failed exit. The freeze/thaw handling uses the helpers from #14305 as well.

Scratch space on the primary storage

Copy-before-write runs inline with the guest's writes and saves every block overwritten during the backup, at most once. The agent therefore creates one LINSTOR resource per backup (cs-bkscratch-<uuid>), with one volume per disk sized for the whole disk plus qcow2 metadata, so the scratch can never fill up and pause the VM, also with several backups running at once:

• The resource definition is created in the pool's resource group, so the group's properties and storage pool selection apply.
• If the VM's host has a matching storage pool, the scratch is a plain STORAGE resource there: it only lives for one backup and needs no replication. Otherwise, e.g. on a compute node without storage, it is a single DRBD replica on another node that the host attaches diskless.
• The agent deletes the resource once the backup has ended, also after a failed or timed-out one and when the agent is stopped normally.
• If the agent dies or the host goes down during a backup, the resource stays behind. When the agent next registers the first pool of a LINSTOR controller, it deletes every cs-bkscratch-* resource definition with a resource on its node. This happens once per controller and agent process: pools are registered again on every reconnect, while backups may still be running. After a hard crash the VM's pull-mode job can outlive the agent and still hold the scratch, so LINSTOR refuses the delete; the warning then names the VM (from Aux/cs-backup-vm) and says to end its job with virsh domjobabort and delete the resource manually.
• KVMStoragePool gets createBackupScratch()/deleteBackupScratch(), implemented by LinstorStoragePool. A pool that cannot provide scratch space fails the pull-mode backup; there is no fallback to host-local files.

NASBackupProvider – routing and cadence

• New decideContentChain() mirrors the bitmap cadence logic (stopped VMs, nas.backup.full.every, chain length) but anchors on backup history alone. There is no host-side bitmap that can be invalidated, so a VM restart, live migration or restore cannot break the chain; content comparison is stateless and self-correcting.
• The master switch nas.backup.incremental.enabled applies first: with incrementals disabled every VM, LINSTOR included, keeps the legacy push-mode full backup. Enabling incrementals is the opt-in to the content-based path.
• ChainDecision separates the two concerns: mode is full or incremental, a new method field is qcow2 or content. Further methods (e.g. an RBD diff) can be added without new mode tokens.
• ChainDecision carries the parent backup uuid directly, since a content chain has no bitmap name to resolve the parent by.
• Mixed-storage VMs (e.g. root on LINSTOR, data disk on NFS) stay off the content path: one backup run uses one mode.
• For content-method backups of running VMs the volume pools and paths are now sent as well (previously only for stopped VMs), so the agent knows where to take the scratch space from.

LibvirtTakeBackupCommandWrapper / agent

TakeBackupCommand carries the new method field, passed to the script as --method. When absent the agent assumes qcow2, so a management server that does not send it keeps working. A content incremental requires only its per-disk parent paths, no bitmap. No new agent.properties keys are needed.

The agent kills a timed-out script with SIGKILL, which no trap can catch, so after a pull-mode timeout the wrapper aborts the backup job itself.

Restore onto LINSTOR: do not skip zero regions (bug fix, independent of the above)

replaceBlockDeviceWithBackup ran qemu-img convert with --target-is-zero whenever the target device existed, skipping every zero/unallocated qcow2 cluster of the backup. On an in-place restore those clusters kept the volume's current data, and a new LINSTOR volume on thick LVM is not zeroed either. A new KVMStoragePool.newVolumeReadsAsZero() hook (false by default, LinstorStoragePool answers it with LinstorUtil.resourceSupportZeroBlocks()) now limits zero skipping to freshly created volumes on thin or ZFS backed pools. RBD targets were never affected.

LINSTOR: wait for the device after making a resource available (bug fix)

createPhysicalDisk and connectPhysicalDisk returned as soon as LINSTOR had made the resource available on the node, but udev creates the /dev/drbd/by-res link a little later. Restoring a backup runs qemu-img convert onto the volume right away and failed with "Could not open '/dev/drbd/by-res/cs-/0': No such file or directory": on every new volume when creating an instance from a backup, and on an in-place restore whenever the host held no replica. Both now wait up to 10 seconds for the local device.

Stopped-VM backups (bug fixes)

• A stopped VM is copied straight from its volume devices on its last host. A LINSTOR volume only has a device on hosts that hold a resource of it, and stopping the VM removes the diskless resource it ran on, so the copy failed unless that host held a replica, always so on setups with separate compute and storage nodes. The agent now makes such volumes available before the copy and disconnects them afterwards; volumes that already have a device are left alone.
• backup_stopped_vm redirected qemu-img convert with > into $logFile, the agent's own agent.log, truncating it on every stopped-VM backup. It now appends.
• After a failed convert the script ran cleanup but carried on with the next disk, which then failed the second cleanup and left the backup in Error state ("Backup cleanup failed"). It now exits after the cleanup.

Behaviour for VMs on qcow2/NFS/Ceph storage is unchanged; they keep using the bitmap path from #13877.

Types of changes

• Breaking change (fix or feature that would cause existing functionality to change)
• New feature (non-breaking change which adds functionality)
• Bug fix (non-breaking change which fixes an issue)
• Enhancement (improves an existing feature and functionality)
• Cleanup (Code refactoring and cleanup, that may add test cases)
• Build/CI
• Test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

• Major
• Minor

Bug Severity

• BLOCKER
• Critical
• Major
• Minor
• Trivial

Screenshots (if appropriate):

How Has This Been Tested?

Unit tests

Seven new NASBackupProviderTest cases cover the routing and cadence decisions and the volume pools sent for content-method backups:

• decideChainReturnsLegacyFullForLinstorWhenIncrementalDisabled
• decideChainReturnsContentFullForStoppedLinstorVm
• decideChainReturnsContentIncrementalForLinstorWithValidParent
• decideChainReturnsContentFullForLinstorAtChainEnd
• allVolumesOnContentDiffCapableStorageFalseForMixedStorage
• allVolumesOnContentDiffCapableStorageFalseForNoVolumes
• takeBackupSendsVolumePoolsForContentMethodOnRunningVm

LibvirtTakeBackupCommandWrapperTest covers the --method / --scratch-devs arguments, scratch creation, sizing and cleanup (also when the backup fails or times out), the method validation, the job abort after a pull-mode timeout and connecting a stopped VM's LINSTOR volumes. The new LinstorStorageAdaptorTest covers local STORAGE-only and remote diskless scratch placement, cleanup of a scratch that cannot be placed, the device wait in createPhysicalDisk/connectPhysicalDisk, and deleting leftover scratch on pool registration (only this node's, once per controller, never failing the registration). LibvirtRestoreBackupCommandWrapperTest covers zero skipping for in-place, new thick and new thin LINSTOR restores.

All NASBackupProviderTest (30), LibvirtTakeBackupCommandWrapperTest (18), LibvirtRestoreBackupCommandWrapperTest (23) and LinstorStorageAdaptorTest (11) cases pass, as do the full agent, KVM plugin, LINSTOR plugin and NAS plugin suites.

End-to-end on a 3-host KVM/LINSTOR cluster (management server + 3 agents, LINSTOR 1.35.2, libvirt 10.0, NFS backup repository, nas.backup.incremental.enabled=true), on the current revision:

Test 6 ran with the small marvin-alpine guest template (see #13746) in a Basic zone. Creating an instance from a backup does not work in a Basic zone on current 4.22 for reasons outside this PR (it passes the backup's network ids, which a Basic zone rejects, and the new instance gets the account's default security group), so those were worked around locally for the run.

How did you try to break this feature and the system with this change?

The first four were run on the current revision, the others on the initial one with host-local scratch files.

• No local storage pool: VM on a host without a storage pool of the resource group. The scratch is placed as a DRBD replica elsewhere and attached diskless; the backup succeeds.
• Scratch cannot be created: the backup fails up front with the LINSTOR error, no backup is started, the VM is unaffected, and a resource definition that was already created is deleted again.
• Agent restarted during a backup: the script's trap ends the pull job and the agent deletes the scratch before it exits; the next backup succeeds.
• Agent killed with SIGKILL during a backup: the pull job outlives the agent and keeps the scratch open, so the cleanup on the next start cannot delete it and logs the manual steps; after virsh domjobabort and deleting the resource the next backup succeeds.
• Missing parent: deleted the parent backup files and requested an incremental. The script falls back to a full, signals INCREMENTAL_FALLBACK=true, and the DB records a full.
• VM restart / live migration between backups: since no host-side bitmap exists, the next backup is still incremental and the delta is correct. This is the case where the bitmap chain has to restart.
• Mixed storage: VM with a LINSTOR root disk and an NFS data disk stays on the legacy full-only path; no half-content, half-bitmap chain is produced.
• Failure mid-loop (killed qemu-img during the pull): the backup job is aborted, scratch and partial output removed, mount released; the next backup succeeds. Deleting the LINSTOR scratch resource after a failed or timed-out backup is covered by the unit tests.
• Legacy DRBD device paths (/dev/drbdN instead of /dev/drbd/by-res/… in live XML): resolved via udev like the push path; if the UUID cannot be derived the backup fails hard rather than producing a file restore cannot find.